jenkins
There are two types of compilation failures: NMM and real*8.

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: 0426150303cab97422a6af0c981c731d0a3aa29a, requested by: dwongepa for PR: https://github.com/wrf-model/WRF/pull/1432. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 19           18
    Number of Builds       : 48           46
    Number of Simulations  : 163           161         17
    Number of Comparisons  : 103           93         0

    Failed Simulations are: 
    output_18:11 = STATUS test_018s em em_real 32 solaraNE
output_18:11 = STATUS test_018m em em_real 34 solaraNE
output_2:2 = STATUS test_002m hwrf nmm_real 34 1NE
output_2:2 = STATUS test_002m hwrf nmm_real 34 2NE
output_2:2 = STATUS test_002m hwrf nmm_real 34 3NE
output_6:2 = STATUS test_006s real8 em_real 32 14
output_6:2 = STATUS test_006s real8 em_real 32 17AD
output_6:2 = STATUS test_006o real8 em_real 33 14
output_6:2 = STATUS test_006o real8 em_real 33 17AD
output_6:2 = STATUS test_006m real8 em_real 34 14
output_6:2 = STATUS test_006m real8 em_real 34 17AD
output_7:2 = STATUS test_007s qss8 em_quarter_ss 32 08
output_7:2 = STATUS test_007s qss8 em_quarter_ss 32 09
output_7:2 = STATUS test_007o qss8 em_quarter_ss 33 08
output_7:2 = STATUS test_007o qss8 em_quarter_ss 33 09
output_7:2 = STATUS test_007m qss8 em_quarter_ss 34 08
output_7:2 = STATUS test_007m qss8 em_quarter_ss 34 09
    Which comparisons are not bit-for-bit: 
    None

These modifications are for facilitating constructing a WRF-CMAQ coupled
model without compromising creating a regular WRF executable.

TYPE: new feature

KEYWORDS: CMAQ, coupled model, constructing, WRF-CMAQ, executable

SOURCE: David Wong, US EPA ([email protected])

DESCRIPTION OF CHANGES:
Problem: A WRF-CMAQ coupled model was developed in US EPA in around 2008. The work
has been published in a journal article [1]. Since then, the coupled model has been
applied to various studies and region to validate the effectiveness of the model [2-12].
Throughout the WRF released cycle, the coupled has been updated manually into a subset
of WRF version. The update process was tedious and error prone. With this PR, the
manual update process can be eliminated.

Solution: N/A

ISSUE: N/A

LIST OF MODIFIED FILES: list of changed files (use git diff --name-status master to get formatted list)

newly added file:

Registry/registry.WRF-CMAQ-twoway
phys/complex_number_module.F
phys/module_twoway_ra_rrtmg_sw.F
phys/module_twoway_rrtmg_aero_optical_util.F

modified file:

Makefile
clean
configure
Registry/Registry.EM
Registry/registry.em_shared_collection
arch/Config.pl
dyn_em/module_first_rk_step_part1.F
dyn_em/solve_em.F
external/io_netcdf/makefile
main/Makefile
main/depend.common
phys/Makefile
phys/module_ra_rrtmg_sw.F
phys/module_radiation_driver.F

TESTS CONDUCTED:

Three different models were built with given lines of instructions:

• model Fix KPP compilation problem due to non-existent empty directory after Github transition #1 (used the original WRF source code without any changes in this PR)

  • went through a regular configure process to build a regular WRF model with the configure command
  • compiled the source code with usual command: compile em_real to generate a WRF executable
    (for discussion purposes, the WRF executable is named p.exe)

• model Fixing typographic errors in the WRFDA code #2 (used the new WRF source code with all the changes in this PR)

  • went through a regular configure process to build a regular WRF model with the configure command
  • compiled the source code with usual command: compile em_real to generate a WRF executable.
    (for discussion purposes, the WRF executable is named np.exe)

• model Removing some debug-level prints from RRTMG lw #3 (used the new WRF source code with all the changes in this PR)

  • issue the following two environment setting commands:

    setenv WRF_CMAQ 1
    setenv IOAPI /home/wdx/lib/x86_64/ifc-18.0/ioapi-3.2-2020220

    (Note: ioapi_3.2-2020220 is another third-party library)

  • went through a regular configure process to build a regular WRF model with the configure command

  • compiled the source code with usual command: compile em_real to generate a WRF executable.
    (for discussion purposes, the WRF executable is named nc.exe)

Case p was based on the executable of model #1, case np was based on the executable of model #2, and
case ncX, where X represents various available options in running the WRF-CMAQ coupled model and detailed
information is provided below, was based on the executable of model #3. All cases were tested
with two days of simulation, 2016/7/1 - 2016/7/2. To verify the correctness of this new PR, a bit-by-bit
comparison of every variable in all time steps on the second day of the wrfout file between two selected
cases.

One of the WRF-CMAQ coupled model running setting, wrf_cmaq_option:

 0 = run WRF only
 1 = run WRF only and           w   producing MCIP like GRID and MET files
 2 = run WRF-CMAQ coupled model w/o producing MCIP like GRID and MET files
 3 = run WRF-CMAQ coupled model w   producing MCIP like GRID and MET files

In these scenarios, X was set to the wrf_cmaq_option and another coupled model
setting, direct_sw_feedback was set to .false., where direct_sw_feedback is a
switch to turn on short wave aerosol direct effect or not.

For an additional case, X = s means

wrf_cmaq_option = 3, and direct_sw_feedback was set to .true.

• comparing case p and case np

  Identical result was rendered. This means that this new PR does not alter anything in terms of
  constructing a regular WRF executable.

• comparing case np and nc0

  Identical result was rendered. This means that this new PR can make the WRF-CMAQ coupled behave
  as a regular WRF model

• comparing case nc0, nc1, nc2 and nc3

  Identical result was rendered. This means that this new PR with or without the chemistry portion
  of the coupled model does not change the WRF result

• comparing case nc0 and ncs
  Different result was rendered with respect to any variables can be affect by radiation calculation.
  This means the aerosol information from CMAQ was transferred to WRF and radiation calculation through
  the short-wave aerosol direct effect.

Here are some of the research work based on the WRF-CMAQ coupled model:

12. Jia Xing, Jiandong Wang, Rohit Mathur, Shuxiao Wang, Golam Sarwar, Jonathan Pleim, Christian Hogrefe,
    Yuqiang Zhang, Jingkun Jiang, David C. Wong, Jimin Hao,
    "Impacts of aerosol direct effects on tropospheric ozone through changes in atmospheric dynamics and
    photolysis rates",
    Atmos. Chem. Phys., 17, 9869-9883, 2017.

13. Wang, J., Xing, J., Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
    "Historical trends in PM2.5 related premature mortality during 1990-2010 across the northern hemisphere",
    Environmental Health Perspectives, Vol 125, Number 3, p400-408, March 2017.

14. Xing, J., Wang, J. Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
    "Unexpected benefits of reducing aerosol cooling effects",
    Environmental Science and Technology, 2016 Jul 19, 50(14):7527-34.

15. Gan, C-M., Hogrefe, C., Pleim, J., Mathur, R., Xing, J., Wong, D. C., Gilliam, R., Pouliot, G., and Wei, C.,
    "Assessment of the Effects of Horizontal Grid Resolution on Long-Term Air Quality Trends using Coupled
    WRF-CMAQ Simulations",
    Atmospheric Environment, Volume 132, May 2016, Pages 207-216.

16. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei C., and Wang J.,
    "Air pollution and climate response to aerosol direct radiative effects: a modeling study of decadal
    trends across the northern hemisphere",
    Journal of Geophysical Research: Atmospheres, Vol. 120, Issue 23, Pages 12221 - 12236, Dec 2015.

17. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., and Wei C.,
    "Can a coupled meteorology-chemistry model reproduce the historical trend in aerosol direct radiative
    effects over the Northern Hemisphere?",
    Atmos. Chem. Phys., 15, 9997-10018, 2015.

18. Gan, C-M., Pleim, J., Mathur, R., Hogrefe, C., Long, C., Xing, J., Wong, D. C., Gilliam, R., and Wei, C.,
    "Assessment of long-term WRF-CMAQ simulations for understanding direct aerosol effects on radiation
    "brightening" in the United States",
    Atmos. Chem. Phys., 15, 12193-12209, 2015

19. Gan, C-M., Binkowski, F., Pleim, J., Xing, J., Wong, D. C., Mathur, R., and Gilliam, R.,
    "Assessment of the aerosol optics component of the coupled WRF-CMAQ model using CARES field campaign data
    and a single column model",
    Atmospheric Environment, Volume 115, Pages 670-682, August 2015.

20. Hogrefe, C., Pouliot, G., Wong, D. C., Torian, A., Roselle, S., Pleim, J., and Mathur, R.,
    "Annual application and evaluation of the online coupled WRF-CMAQ system over North America under AQMEII phase 2",
    Atmospheric Environment, Volume 115, Pages 1-756, August 2015.

21. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei, C., Gilliam, R., and Pouliot, G.,
    "Observations and modeling of air quality trends over 1990-2010 across the Northern Hemisphere: China, the
    United States and Europe",
    Atmos. Chem. Phys., 15, 2723-2747, 2015.

22. Wang, J., Wang, S., Jiang, J., Ding, A., Zheng, M., Zhao, B., Wong, D. C., Zhou, W., Zheng, G., Wang, L.,
    Pleim, J. E., and Hao, J.,
    "Impact of aerosol-meteorology interactions on fine particle pollution during China's severe haze episode
    in January 2013",
    Environ. Res. Lett. 9, September 2014.

23. Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., and Kang, D.,
    "WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results",
    Geosci. Model Dev., 5, 299-312, 2012.

I have not done a WRF PR before. Please let me know what kind of specific test I have to perform. Thanks.

@dwongepa
David,
Would you build the standard ARW code with a special flag on the configure command:

./configure -r8

Then do everything else as usual. This will build the code with 64 bit reals. Those seem to be failing right now. Once you get the compilation working, go ahead and push those mods to the repository so that we can see what the automated testing status is.

@dwongepa
David,
You report 18 files changed, but github finds 19. Would you go through the list and update accordingly.

@jordanschnell
Jordan,
Just want to put this on your radar

@dwongepa
David,
Here are the results of the jenkins tests:

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: 8b48332731251149548b864e75f7abda44d29ba0, requested by: dwongepa for PR: https://github.com/wrf-model/WRF/pull/1432. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 19           18
    Number of Builds       : 48           46
    Number of Simulations  : 163           161         14
    Number of Comparisons  : 103           93         0

    Failed Simulations are: 
    output_18:11 = STATUS test_018s em em_real 32 solaraNE
output_18:11 = STATUS test_018m em em_real 34 solaraNE
output_6:2 = STATUS test_006s real8 em_real 32 14
output_6:2 = STATUS test_006s real8 em_real 32 17AD
output_6:2 = STATUS test_006o real8 em_real 33 14
output_6:2 = STATUS test_006o real8 em_real 33 17AD
output_6:2 = STATUS test_006m real8 em_real 34 14
output_6:2 = STATUS test_006m real8 em_real 34 17AD
output_7:2 = STATUS test_007s qss8 em_quarter_ss 32 08
output_7:2 = STATUS test_007s qss8 em_quarter_ss 32 09
output_7:2 = STATUS test_007o qss8 em_quarter_ss 33 08
output_7:2 = STATUS test_007o qss8 em_quarter_ss 33 09
output_7:2 = STATUS test_007m qss8 em_quarter_ss 34 08
output_7:2 = STATUS test_007m qss8 em_quarter_ss 34 09
    Which comparisons are not bit-for-bit: 
    None

The failures are ALL related to real8 issues. My guess is that there are some real8 inconsistencies that are breaking the compile. You can test this with

clean -a
configure -r8
compile em_real

@dwongepa No need to worry about NMM compile. For those commented out state variables, would you consider packaging them for CMAQ coupling? That way they also won't show up in a regular WRF run.

@dwongepa
David,
There are a few small modifications that are required. One of the WRF model options is to run all computations in real*8 precision. Because of this, the values such as 0.0D0 are now 16 bytes. This causes inconsistency in your internal routines.

module_twoway_rrtmg_aero_optical_util.f90:1690:10:

 1690 |      II = c_set(0.0D0, 1.0D0)
      |          1
Error: Type mismatch in argument 'x' at (1); passed REAL(16) to REAL(8)
module_twoway_rrtmg_aero_optical_util.f90:1732:14:

 1732 |      chi0y2 = c_mul(-1.0d0, c_SIN(y2))
      |              1
Error: Generic function 'c_mul' at (1) is not consistent with a specific intrinsic interface
module_twoway_rrtmg_aero_optical_util.f90:1734:14:

 1734 |      chi0x2 = c_mul(-1.0d0, c_SIN(x2))
      |              1
Error: Generic function 'c_mul' at (1) is not consistent with a specific intrinsic interface
module_twoway_rrtmg_aero_optical_util.f90:1739:14:

 1739 |      xback  = c_set(0.0d0, 0.0d0)
      |              1
Error: Type mismatch in argument 'x' at (1); passed REAL(16) to REAL(8)
module_twoway_rrtmg_aero_optical_util.f90:1793:22:

 1793 |               brack = c_set(0.0D0,0.0D0)
      |                      1
Error: Type mismatch in argument 'x' at (1); passed REAL(16) to REAL(8)
module_twoway_rrtmg_aero_optical_util.f90:1794:22:

 1794 |               crack = c_set(0.0D0,0.0D0)
      |                      1
Error: Type mismatch in argument 'x' at (1); passed REAL(16) to REAL(8)

The WRF model has an ifdef available to determine if the default wordsize is 4 bytes or 8 bytes for reals. Here is an example of how to use it:

#if ( RWORDSIZE == 8 )
      REAL        , DIMENSION(:,:) ::   dbl2d  
#else
      REAL(kind=8), DIMENSION(:,:) ::   dbl2d 
#endif

@dwongepa @weiwangncar
David,

1. I put in the mods to get the code to build with real*8 default floats. (Yay for small successes!) Check my last few commits.
2. With a nested simulation, when I turn on the physics option swint_opt=2, the code generates NaN values in the SWDOWN2 field (only in the CG field).
3. When I turn off the nest, the NaN values are still present.
4. When I use swint_opt=1 or swint_opt=0, the output from the WRF executable is fine.

@dwongepa @weiwangncar

Forget the previous "all OK" note.

This namelist option combination causes NaNs in SWDOWN2 with ncdump wrfout_* | grep -i nan:

 &physics
 swint_opt                           = 2,
 ra_lw_physics                       = 4,     4,     4,
 ra_sw_physics                       = 4,     4,     4,

@dwongepa @weiwangncar @pedro-jm @dudhia
Here's a picture of d01 SWDOWN2 with the CMAQ coupling mods. When using RRTMG and swint_opt=2, the simulation gets NaNs. The NaNs are in row 1, at each and along only those columns that have non-zero SWDOWN2. Does this ring any bells for anyone?

@dwongepa @weiwangncar @dudhia
David,
I re-did the PR, using the latest develop branch.
Please conduct a couple of tests ON THE OTHER BRANCH to make sure that the coupling functionality is still working.

I am closing this PR in deference to "CMAQ-WRF coupling: cherry pick" #1609